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ABSTRACT ^ 

The educational audit is suggested for assessing the 
process of inquiry for reliability and the product of x inquiry for 
absence of bias. The inquiry" auditor must review the inquiry 
processes to determine that they conform to riorms of *good 
professional practice* 1 * He* must review inquiry products "to ensure 
they ban* be substantiated from data collected. These tasks are 
equivalent to establishing the dependability of an inquiry and the 
conf inability of its data and conclusions. The inquiry audit process 
is composed of the audit trail and the audit procedure. The 
documentation for a naturalistic audit trail incorporates a reflexive 

'journal and a detail of procedures. The audit procedure consists of 
several basic steps. These include the decision to do the audit; 
acquisition of the inquirer's report a*id all portions of the audit 
trail; determ* ration of documentation sufficiency; comparison of 
procedures to problems addressed; comparison of raw data to written 
narrative; description of comparison results; noting changes in 
jnethods of deployment £>f personnel ai^d judgments about context and 
problem; noting whether inferences flow logically from data; and 

0 certification of findings in final isport. An .audit procedure is 
presented in diagram. (DWH) 
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INTRODUCTION 
Naturalistic inquiry, has experienced an upsurge in interest, 
and judging by program entries In *?uch national meetings as the 
American Educational Research Association, the Evaluation 
Network, and the Evaluation 'Research Society, in application also. 
Nevertheless, detractors of .the paradigm still view the results of 
such inquiry as untrustworthy and potentially biased. Because 
the paradigms res tar on such beliefs as: that the Inquirer himself 
is the most useful instrument, that the inquiry design cannot be 

♦ 

specified a priori as in typical experimental . design, thaf the 
guiding theory is itself grounded in emergent dara, and that the 
preferred inquiry setting is natural situations and contexts 
(which are uncontrolled) rather than the laboratory (highly 
controlled), naturalistic inquiry continues to be viewed as un- 
disciplined, inexact, and highly subjective. 

Guba (1978, 1981), Guba and Lincoln (1981), Lincoln and 
Guba (1981), and Lincoln (1981) have made the case that all four 
of the "trustworthiness" criteria that have been posed tradition- 
ally for inquiry can be met by naturalistic inquiry as well, albeit 
in somewhat redefined form—a form consistent with assumptions of 
the naturalistic Raradigta. Thus, we have argued that the con- 
cept of internal validity should be replaced by that of credibility, 
external validity by "transferability, reliability by dependability 1 , 

1 Dependability, according to Guba (1981), is a concept that 
accounts both for unreliability in the conventional sense as well 
as for other shifts or changes that are deliberately introduced by 
thef investigator as tlie design of the inquiry emerges or unfolds- 

» 
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and objectivity by conftrmabiliiy . Specific techniques thai ihe 
naturalistic inquirer can use to assure a sound level of perfor- 
mance with respect these criteria have been proposed. 

A major technique suggested by ua Is the so-called educ«- 
Uonal audit, which we recommerid both for assessing the process 
of inquiry for reliability (a dependability audit) and H the 
product(s) of inquiry for absence of bias (conflrmabtllty audit). 

The emphasis on both process and product Is a crucial one, 
we believe. In traditional forms of rationalistic inquiry, choice of 
proper procedure (processes) virtually Guarantees results {Inquiry 
products) which are trustworthy or unassailable. When the 
criteria of internal validity, external validity, reliability and 
objectivity or neutrality have been met, then the results are 
presumed to be trustworthy (or rigorous). But the blurring of 
the distinction between processes ajtd product has allowed attacks 
on naturalistic inquiry to proceed without requiring careful exam- 
ination of whether in fact, the* attacks are justified or reasonable 
ones. That point, that "there is both a difference and a distino 
% tion, has been made by Cronbach and Suppes (1969), who assert 
that the feature which most prominently distinguishes disciplined 
inquiry from other forms is that it be c onducted (the process) 
and - reported (the product) in such a way that all Of its aspects 
can be publicly examined: 

. . . the report of a disciplined inquiry has a texture that 
displays the raw materials entering the argument and the 
16gical processes by which they* were compressed and re- 
arranged to make the conclusion credible (p. 16).. 

The ^udit is suggested as a means for carrying out the kind 

of public examination suggested by Cronbach and Suppes. The 
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use of l the term "audit" in po Accident, tt should be noted, but b 
based ppqn the flacal Audit aa 3 metaphor. According i« Wheat 
and others (1972, p. 14), the uoala of a Hacal eudit are threefold 
1) to discourage practices in specif U* areaa which experience 
Indicates might be employed in such a way as to mislead public 
investors; 2) to encourage practices which could be expected to 
make financial statement** more informative; and .1) lo reduce the 
use of alternative accounting methods not Justified by factual or 
circumstantial differences. Tor the (local auditor, called In to 
examine the accounts of a client corporation, achievement of these 
goals translates into two majgr tanks. 

His first responsibility in to examine the processes by which 
the local accounts ara*kept, not so much to assure that thore hes 

^ I v 

been lio fraud (although it <s expected that fraud will be detected 
if has occurred) than to assure that the oo0k& represent a "fair" 
statement of the company's position. ~J2\t6 auditor is particularly 
concerned that there should have been no !, creative accounting 11 
Uiat makes the company appear more solvent than it Is, for the 
sake, for example, of attracting investors. The auditor's major 
interest is that he is able to certify that the processes used fall 
within the bounds of good professional practice. * 

His second responsibility is to examine the products of -the 
local accounting process, to ascertain that every entry in the 
books can fcje authenticated by documentation oc by solicited 
confirming > statements, that the "bfcttom line M is correct, and that 
interpretations made of the accourits in any fiscal statement based 
on them are accurate and appropriate. 
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In similar ^ashion, the inquiry auditor has two task* to 
review the inquiry processes to ha certain that they tali within 
the norms of "goad professional practlsa' 1 , and to review the 
inquiry products to he certain that tttiy can be substantiated 
from the data collected, in other work, w* have as»erted, the 
former task Is equivalent to establishing the de^andabJUly of an 
inquiry and the latter the conffrmabllfty V of its data and con- 
clusions. The issue, it should be noted, is not whether the 
lnvefilloator carried out the processes or reached tins conclusions 
in 'the name way that the auditor would have, hut whether they 
were carded out in ft reasonable manner. Thus, •replicabiilty is 

nW a criterion, but rather rationality Is, In this context, 

\ * 
"reasonable 11 and "rationality" are taken to mean that the methods 

chosen for data collection are appropriate to the problem to be 
studied; that the techniques of analysis utilized arc those con- 
sonant with the form In which data are collected and assembled; 
that reports of the date arc coherent, credible, and exhibit 
structural corroboration; and that all assertions about the con- 
text (save for the inquirer interpretations) may be traced to 
authentic data units or categories. 
Why Create an Audit Trail or Perform an Audit ? 

Since so few studies, save in the physical sciences, are 
replicated or have results which are re-examined, why would one 
want, to create an audit trail or perform an audit in the first 
place? We would suggest three reasons. 

* First, the issue of trustworthiness must be addressed* 
When a scfentist performs an experiment (engages in inquiry), 



the materials, method* and choice of variables to be studied are 
catalogued cleanly as part of the reporting of Jhe study itself. 
The format for this reportage is fairly well accepted, and any 
contributor to a major professional conference Is often .confronted 
with advice to construct his proposal, and presumably the com- 
pleted paper, along a certain accepted format, which Includes 
first, a statement of the problem, second a statement of objer 
tlves, third a statement of tha methods, and so forth Journal 
articles more often than not follow the same pattern Thus, if a 
proposal or article is well done, the variables, procedures, 
methods And conclusions fohow from a statement of the problem. 
Thus, the research reporting has left its own audit trul), and 
presumably, by choosing the same population (or sample thereof), 
the sam<< variable.^, instrumentation and methods, a second scien 
tlat should be able to replicate a colleague's findings. 

The posture of rationalistic researchers toward naturalistic 
researchers is in part understandable, since procedures and 
methods for carrying out scientific research have evolved za 
clearly over so long a period and since, as Mautz and Sharaf 
point out, .procedures and Methods become "peculiar" to dis- 
ciplines and become integral to the search for data: 

"If ohe carefully observes the methods followed in different 
disciplines , he will discover that each has developed hn * 
attitude and procedure peculiar to itself- Some of 'these 
approaches have important characteristics in common, but 
there ara -"fclso significant differences. As each * discipline 
develops into maturity, it continually experiments and modifies 
it3 procedures and attitude until it finally devises a method 
appropriate to its activities- The method of inquiry thus 
becomes as much as integral part o£ the discipline as does *he 
subject matter itself" (1961, p. 18). 
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In the social sciences, we have not always followed this 
procedure in reporting. Our studies are noy clearly experimental 
(nor should they be); e*jr reports have no accepted formats, 
some being presented In case study form, our populations have 
been often unique pnd most assuredly not typical In any sense 
assured by randomization (which Ja totally appropriate;, and we 
have relied upon our Insight and imputations to "guarantee 11 the 
trustworthiness of * our findings. But experimentalists h**ve 
mounted concerted attacks upon naturalistic Inquiry, over the 
issue of trfistworthin&ss, which they label rigor. Tor that 
, reason, the audit trail which is constructed, and the audit pro- 
cess which Is performed allows for the work of naturalistic iriqulr- 

« 

ers (and others who prefer to tail themselves by other names, 
but whose work Is phenomenoiogkal In orientation) to fulfill the 
criterion of public lnspectabillty . 

A second reason for calling for audits can be found In the 
numbers of researchers who are calling foi^a decrease in emphasis 
on the creation of data de novo (Lincoln, 1978; Guba and Lincoln, 
1901 ; Burstein , ]L978) and for the secondary analysis , re- 
analysis, or meta-analysis of existing data banks or scarcesf 
'(Lucas, 1974a, b; Glass, 1976). The use of in-place, existing 
resources to sefve new purposes or meet new research ne^ds Is 
an idea both born of need and inherently sensible, and tech- 
niques are available to perform new kinds of analyses of these 
purposes. Thus, the creation of an audit trail in effect creates a 
data bank of oral and documentary history for future researches 
to reanalyze in light of new questions, 
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Third,, find closely related to the second abpve, is hat the 
produce of naturalistic and ethnomethodologlcal research form a 
valuable jbody of records "on any project or context. Au a resyit, 
gome researchers, notably Stenhouser (1978) are ceiling for the 
creation of archives which effectively mark when and where 
researchers have been, and what their effects might be. Thus, 
the creation of an audit trail in effect assures a data bank of oral 
a documentary history for future researchers to reanalyze In light 
of questions v/hich haVe not yet been framed, 

The Inquiry Audit Process 

Whale previous authors have laid out the auditing t^sk and 
suggested its major conceptual parameters (Spradley, 1981, Cuba, 
1978, 1981; Halpern, 1981; WUIower, 1981), few specific sugges- 
tions have been offered about how' to parry it out. Specifically, 
two major questions must be addressed; 

1. What is the nature of the audit trail which must left by 
the Inquirer, analogous to the accounts left by an accountant, 
which foftn the basis for the later audit? This problem is 

4 

typically unaddressed in fiscal auditing literature since methods 
for keeping accounts ara v/ell understood and taken for granted 
in fiscal operations. 

2. What are the steps that the auditor must take in 
carrying out an audit? How do these steps differ for the depend- 
ability audit and the confirmabiiity audit? 

It Is to these two questions that the major portion of this 
paper will be addressdd. 
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The Audit Trail 

Guba (1981) h#s admonished adherents to naturalistic para- 

digm research that; 

M * * _ • the ^hatJrailBt wtiL during the study establish an 
'audit trial* that will make it possible for an externa! auditor 
to examine the processes whereby data were collected and 
analyzed and interpretations made. 

After completion of the study the naturalist will arrange fo0 
a dependability audit to be done by an external auditor-- 
someone competent to examine the audit trail and to comment 
upon the degree to which procedures used fall within 'gener- 
ally acceptable' categories. Such t an audit is concerned 
primarily with process." (p. 20) . . .\ 

[and] after completion (St the study naturalists will arrange 
/ for a iconllrmability audit certifying that data exist in 
' support of every -interpretation and that the interpretations 

have been made in ways consistent with the available data. 

(p. 88) 

% 

The documentation for such a trail " incorporates in somewhat 
expanded form two standard and well-khown research tools The 
first is the field journal or field logs (workbooks', journals and 
field notes)* of the anthropologist or field sociologist, the second 
is the detail of procedures kept by the laboratory scientist Both 
of these foiws of documentation are appropriate for the natural- 
istic inquirer albeit recast In slightly different form* Elements of 

each, more formally structured, go into providing the documents- 

* / 

tion necessary for either a product or process audit, 

hi least six different forms of archival materials need to be 

maintained. These include: 

all raw data, including interv^w and observational 
notes, test scores, documents *d records collected in 
the field,- records 61 unobtrusive measures made, and 
the like; 
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a lod of all activities, including all field contacts with 
d&tes^ purpose, substance and outcome noted; 
a log of. all methodological decisions made which Influ- 
enced the final emergent design of the Inquiry; 
a log of all, data analysis activity; since it Is assumed 
that data analysis will be a^tinuaily* ongoing and that 
early analysis will affect later steps, it is imperative to 
be able to track the unfolding * alysife alWn the fact, 
a reflexive diary which records the inquirer's own 
perceptions, changing inughts, effective response'!, ih% 
"experiences , ideas, fears, mistakes,, confusions, 'break- 
through^, ^nd problems that aris<* Iduring the ffcld 
work 11 (Spradley, 1979, p. 76), as a partial means for 
providing checks on the e valuator's , own biases; and 
a log of professional contacts that may influence the* 
evaluation, for example, debriefing sessions with nomn- 
volved professional peers"- fLtacoln and Gub^t, 1981). 
The two elements of this trdi t&at may require additional 
explanation are the reflexlVe diary and the log, of professional 
contacts, since both of those are crucial to tracing biases which 
may be purported to creep into a study which is naturalistically 

■v 

oriented; \ ' 

The reflexive journal is analogous to the anthropologists' 
field journals and is the major means for an inquirer to perform a 
ruhriing check on the biases. Which he carried wfth him Into the 
context, * Lincoln and Guba h«r/e previously described in detail 
what forms of material ought to^ be kept in a reflexive journal. 



Dlvftfed into five section^., the Inquirer 1 personal Journal, which 

will later form *a part of the audit tr^tl, ought to" contain the 

following sectiohs* , 

log of evblvinq perceptions . This log beging 
with the Written perceptions of the evaluator prior to hRr 
v entrance- on site.. The ©valuator writes dbyn everything he 
; or ohe Relieves will be found at.the site or 1 program, context. 
.Glearly., what he writes down must be a function entirely or 
his A priori perceptions "(which may or may not be biased, 
although It is not bias which k the Issue at this point). 
This experience should be repeated at regular intervals 
which are function of the time to be spent at site and the 
detail "Jevfel of the writing. The separate renderings may be 
kept in escrow by someone else— usually- an auditor or peer 
. debriefer— but at least sornfeone who Is not directly connected 
with the program oh project evaluation. The writer will want 
to attest that be does not return to earlier versions as he 
writes later ones* At the end of the prgject Or during Its 
duration if ft is oi sufficient -time lapse), these perceptual 
✓ statements 'may be compared to one another and/or to aspects 
of the design and/or analysis to check whether learning Is 
occurring^, ^whether original perceptions and beliefs" persfever- 
ate, whether later findings are clouded by these percep- 
tions, and. whether or not the evaluator has "learned 11 any- ' 
thing tt new H or changed himself, -* 

2. A log of day-to-day procedures . The? purpose of 
~"this log Is to Indicate *In diary or chronograplitc form exactly 

what was .accomplished every day. The log, we would 
recommencj, ^should be t kept In m bound book so that it can 
be verified that no pages were added or^deleted. * Pages 
should be dated and .initialed by the .writer, *amd if it aeeras * 
appropriate, the dating and initialing can be .witnessed. The 
most Important use .of this component of th£ Journal will be 
in the civjalu&tion audit process. 

3. K log of methodological decision points . This 
particular log of £he reflective Journal is entered on an ad 
hoc basis as necessary, and should recorl all major ^nethocP 

. ologlcal. decisions, such as explicating nwct'deslgri steps 
* decisions on instrumentation, flnalizatJon of an analysis 
category set or the like, and such jiec^ons should, be 
entered together frith * reasons or^ rationale for the action 
taken or decision made, These pages also should be dated 
and initialed by the evaluates or eyaluation team director and 
may also be witnessed. The auditing process is the. chief 
purpose for this log. 

4- A log of day-to-day personal Introspections . Sam 
Seiber recommends, that a log (or fieid diary) of this sort of * 
introspection be kept In any type of field work In any of the 
Social sciences. Here one lays out in diary form one f s own 
thoughts and feelings, including stresses one Is ahsost 
bound to undergo (Zigarmi and Zigarmi, 1978) and frustra- 
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tlons one encounters, and- how .these feelings and situations 
are perceived to be changing. Numbers* 2 and 3 above may 
be thought of as ways of representing metftod and" numbers 1 
and '4 amy be thought of as ways of representing persons 
who do inquiry and evaluation. It is important to have some 
insight? into one's self. and to work on generating that in- 
sight. One's willingness to take a situtation on . its own 
terms, for example, depends heavily on whether one is at 
ease with It or not, whether one is encountering frustration 
at not being able to acquire good information, whether one 
suspects he is beipg lied to, and the like. 

These entries should also, be made 'in bound books, 
although Sieber has recommended loose-le&f, notebooks, and 
-should be date*d and initialed. The chief use to which these 
personal logs can be- put. are to test for bias in the evaluator 
and to relate decisions about design jand procedures to it 
later. These logs, however, may, also be utilized to indicate 
to the evaluator where he might need additional training 
(e.g., in Interviewing reluctant interviewees), or wheYe he 
might want additional work beiore undertaking another 
evaluation (e.g., in public speaking and presentations, in 
testing, in negotiating a contract, and* the like). 

S. A log of developing* insights and hypotheses . The 
emergent design wiii.depand almost exclusively on how the 
evaluator takes advantage of what he has already learnud. 
The purpose of this section is to keep readily available an 
up-to-date summary of where one is with respect to know- 
ledge of the situation and working hypotheses about it. 
This is a working section of the journal, which should be 
kept in loose leaf binders and updated as needed. Those 
working hypotheses which have been ( discarded or "out- 
grown" ought to be relegated to historical files oh the pro- 
ject, and retained as part of the audit trail. * 

Entries in the log should be cross-referenced to original 
data. As the evaluation progresses, a section of this log 
can be devoted to a listing and explication of possible items 
to be tssfcel with\*raraerous data sources — a process often 
called member checks --.for the sake of determining cred- 
ibility. It should ne possible to relate, at any time, what is 
In xh\a log |© other fileS^of both raw and processed data. 
There ara several ufees for this log, Including the guiding of 
the Inquiry, shapind/ of the emergent, design, providing the 
basts for subsequen/t dftta collection, and' analyzing activities, 
and for post hoc /auditing procedures (Lincoln and Guba, 
1981, pp. 10-13). j 

We would like to Ipoiiit out that while essentially similar 
records are necessitated* for the inquiry as a whole , when the 
effort l j to carried'out by two persons cr a team of inquirers, 
evolving perceptions, day-to-day procedures, methodological 
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decision points, personal introspections and developing insights, 
and hypotheses are likely to be different from inquirer to in- 
quirer. Because inquirers are likely to work -alone for some 
periods of time between team consultations on emerging d^ign 
issues, the tasks-iji which each is engaged may be suostantiajly 
different and the persons whom one interviews will often provide 
varying insights and hypotheses. As a result, substantially 
different reflexive journals may be kept. Each of these con tri- 
butes, however, to the final audit trail which is established. 

Debriefing sessions, whereby field researcher check in- 
sights, hypotheses and developing theories with peer profes- 
sionals external to the inquiry effort, may be carried out on an 
individual or team basis, or both, when performed op an individ- 
ual basis, the reflexive journal would caVry accounts of the peer 
debriefing; when performed either on a team basis, or between 
teams operating in split-half fashion, (see Gub: . 1981; Lincoln 
and ckiba, 1981; and forthcoming), the audit trail assembled by 
the team- leader would contain records of the debriefing session. 
In any event, the purpose of such debriefing is to keep the 
inquiry effort, including data collection and interpretation on 
track and grounded Ir> the context. 

* Other professional contacts which may be related to the 
research or evaluation effort should also be recorded. Such 
contacts might include, for example: conversation with other 
inquirers engaged in similar research; contacts with other evalua- 
tors evaluating projects which appear similar; or conversation 
with other i social scientists (such as sociologists, anthropologists, 
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or political scientists) who may have operated in the same, or a 
•similar, context. The point is that not all professional contacts 
need * debriefing sessions; some may be information-gathering 
excursions and some may be simply "shop talk 11 concerning the 
progress of the* research. But fairly complete records should be 
kept of these encounters. 

Halpern (1981, p. 12) suggests that audit trails are com- 
prised of "three general classes" of evidence. Those include: 

"(1) natural evidence; (2) created evidence; and (3) "rational 

i 

argumentation". And Mauti and Sharaf (1961, p. 60) argue that 

s 

there is evidence in the world around us (natural) ; evidence 
requiring some effort to bring it out (created or experimental); 
and evidence in the form of ideas which 'flow logically *from ob- 
served facts 1 (rational^ 1 . We would argue that the process 
auditor might be primarily, though not exclusively, interested in 
natural evidence and how it was garnered. The produc t 
(confirmability) auditor might be somewhat more interested in 
created evidence (influences, working hypotheses, and the like) 
and in rational evidence (coherence, structural corroboration, 
wholistic properties , internal congruence, fidelity to member 
experiences, contextual isomorphism,, and believability). 

The focus of an audit, as explicated earlier, is baSed on the 
rationality of data collection methods and analytic teaching and on 
the structural cogence o£ the final report; and on ti^ reason- 
ableness of the conclusions on findings. The audit procedure is 
somewhat less clear . Halpern summarizes the situations aptly : 
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"Data scrutinizing ... is the main task of the auditor 
... The data may have been gathered in many different 
ways. The auditor is privy in most instances only to 
second-hand 'data—those 'communications 1 whirfi^have 
already "-been decoded and recoded by the inquirer. 
Thus, from a wide array of interactions experienced/ by 
* the inquirer, the audit trail is only as close t</ the 
phenomenon as the inquirer's collection procedures 
permit./." (p. 27). J 

Since the auditor is removed from some, much, or all of this 

first-person, original data, procedure* for the audit becomes 

important. While we .perceive no necessity for a single picscrip- 

tiort to* fit all inquiries, certain steps probably constitute a mini- 

MliuiT^.prQ nft dvre . ' Those are designated as steps toward auditor 

. warranty. 

In general, in performing the actual audit, the auditor, 
presumed to be a professional peer of the inquirer, may perform 
either or -both the dependability and confirmability audits. In the 
role of dependability auditor, he will examine all of the documen- 
tation from the point of view of its acceptability within the norms 
of good Naturalistic professional practice . Upon completion of 
that task, he will certify that the "inquiry has been ^adequately 
^and fairly executed from a methodological point of view, probably 
issuing a formal statement to that effect which mai be attached to 
reports of the inquiry, for example, when they are submitted to 
journals for publications. 

In the role of confirmability auditor, he will examine the 
analyzed data, comparing some sample against selected Original 
data items - for example, interview notes or documents - to 
satisfy himself on several counts. First, the data iteqis should 
have been reasonably unitized (that is, reduced to the smallest 
units of measuring required by the problem and consistent with 
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their original form). Second, the units should have been reason- 
able categorized into an appropriate category system (or a tax- 
onomy if the t data permit) , and that individual data items have 
been reasonably assigned to taxons or categories. Third, con- 
elusions should be documentable -in^ terms of the categorical 
system. Finally, conclusions should be demonstrably triangulated 
by reference to multiple data sources (preferably collected and 
analyzed by multiple methods and representing multiple perspec- 
tives). When he has completed this task, the auditor will certify 
that the inquiry products are properly founded on the data and 
have been reasonably interpreted from^them. 

Specifically, however, getting to the auditor warrant is a 
series of "decision points, any one of which may be a go-no go 
decision; To reiterate, there is no single "right" procedure, but 
some basic steps may be drawn. 

The Audit Procedure 

Basically, the audit procedure is carried out in eight steps. 
While other steps may be inserted into the procedure as deemed 
necessary by the inquirer, f under, sponsor, or auditor, these 
steps are those necessary and sufficient to initiate and conclude 

r 

an audit. 

* * 
Step 1; Decision to do audit . Either because of contractual obli- 
gations, or because the results of an inquiry are under fire, or 
because of some other reason (e.g., simply for purposes of 
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external validation), a decision may be made to perform an in- 

Ik 

quiry audit. For .whatever reason it is undertaken, * an audit may 
be commissioned by any one of a number of parties, and it is the 
^ responsibility of the inquirer to maintain ~ and^in legitimate 

- cases, turn over adequate documentation and raw data to 
substantiate his findings and conclusions. 

Who may commission an audit is not entirely clear, although 
a preliminary guess would include the inquirer himself, as a 
means of verifying that his processes, procedures and products 

\ are professionally sound; the funding agent or sponsor of the 
inquiry in the. case of an evaluation (a meta-evaluation) in the 

\ original contract; or other agencies or groups who have a right 
t^o the data and who may question the conclusions or recommen- 
dations. Included in this last group might be, for instance, 
parents of children in ja program which was found to be 
"non-cost-effective" and therefore discontinued, or members of a 
targeted group whose programs were being cut. These persons 
are known elsewhere as stakeholding audiences. 

Step 2: A cquire inquirer 's rep ort and all portions of audit trail . 
It is presumed that field notes have been suitably coded so that 
identification of individuals is difficult, if not Impossible, but 
otherwise, the auditor should have access to field notes, reflexive 
journals, and all other raw data which have been collected And 
which have gone into the final analyses, Upen negotiation of a 
suitable auditing contract, these documents, films, tape record- 
ings, transcripts and the like should be turned over to the 
auditor, who will perform Step 3, \ 

18 



17. 



Step 3: Determine' whether audit trail is sufficiently complete to 
perform audit . A thorpugh inspection of the data ,and products 
of the inquiry should be carried out to determine whether, in 
fact, sufficient documentation is present to constitute an audit 
trail. This is essentially a "no-no go" decision, for if the docu- 
mentation doesn't exist, an audit cannot be performed. 

Step 4-A: Compare procedures to probiem(s) addressed . The 
purpose of this step is to di over whether the inquiry problem 
was one whi£h suitably might be addressed by naturalistic in- 
quiry* and whether a naturalistic inquiry was in fact carried out. 
While we do not wish to re-open the qualitative-quantitative 
debate which has been raging for some time, it is Important to 
determine whether the study is a naturalistic one or fundamentally 
a rationalistic paradigm inquiry, carried out through utilizing 
qualitative methods. A study of the second sort is not, ana 
should not be, amenable to this form of inquiry audit processes. 

m 

Stqp 4-B: Compare raw data to final product (wri tten narrative) 
aft ft check yintUzlnq and rategorl&ing syst e ms and la belling pro- 
cedures. Thc N purprac of this aub-$tep Is to determine whether 
the analyses of data followed consistent and uniformly applied 
rules (which should be provided to the auditor) for unitizing 
items of data, for categorizing or t^xonomtsring those data, and 
for assigning labels to various categories and aggregates of data 
units. Data* units should be clearly distinguished from one an- 
other, ^nd categories ought to deaplay holistic qualities which 
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suggest tha boundaries of the inquiry and context of the inquiry 
alike, , 

Step 5: Describe the results of both comparisons in Step 4 . The 
results of this step should yield information in at least three 
areas: inconsistencies between problem and method, or between 
data and final "product; information about possible inquirer bia? 
which is not made "explicit in the final report or inquiry product; 
and information abo\it the rigor and mutually exclusive or 
non-exclusive nature of the categorizing system used in data 
analysis. 

Step 6: Note shifts in methods deployment of personnel, and 
judgments about context and problem . Since emergent designs 
often characterize naturalistic inquiries, shifts between proposed 
activities and lines of 'inquiry, as well as shifts in insight from 
previously held convictions, should be noted lA this portion of 
the auditor's report. The question to be addressed is most 
appropriately whether or not the shifts were supportable 'or 
sensible in view of the data collected, in view of the insights 
gained, or in view of the biapes exposed and examined. The 
nature of such inquiries to unfold, , rather than to be pre- 
ordained, demands that such shifts be fully documented and that 
the payoff in terms of data be clearly recognized. Since such 

i, 

l 

shifts are on expected part of the research, each should occupy a 
portion of the field logs. 
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Step 7: Note whether inferences flow logically from data . ' A 
significant portion of the product warrant is certification of 
whether the conclusions reached by the inquirer are in fact 
warranted by the data. It is possible, for instance, to have good 
raw data, and sound methodological processes, and to draw un- 
warranted conclusions. For that reason, it is jnperative that the 
auditor note whether the conclusions drawn by the data gatherer 
make sense ^ flow logically from the data, or cohere to form a 
meaningful whole to an outside observer. 

This is an especially sensitive step) since we (and many 
others from disciplines such as anthropology amd sociology) have 
argued for some tiirife that the data analysis process is essentially 
artistic one, whereby the inquirer brings to bear both pro- 
positional and tacit knowledge in, interacting with his data in 
order to fnpve to a level of meta-analysis, to move beyond the 
data to insight and to n\eaning^ has not been stated pre- 

viously. While it is clear that an auditor rnay be able to follow 
the trail of processes, and should be able to track data through a 
study, it is not clear that he will be, able to bring to bear the 
insight, judgment and wisdom that th- original inquirer brought 
in reaching his conclusions. But he should be able to certify 
that those conc lu sions appear logical *nd sensible from the data 
a nd processes outlined * 

Step 6: Certify In final report what ia found . A final audidng 
report should be formulated (or, in the case of separate process 
and product audits, two reports),* which describes in detail what 
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the auditor has found in tracking the study. It should note' 
whether both processes and products appear to conform to reason- 
able canons of good naturalistic practice and whether structural 
coherence exists* in the Report. It should contain some summary 
judgment as to whether conclusions (and recommendations, if they 
are part of" the inquirer f s final report) appear to be warranted 
from the raw data, and should note the extent to whiGh the final 
report gives a vicarious experience to the reader via thick de- 
scrip tion and strongly focussed narrative. 

The. * steps appear to us to be the necessary and logical 
ones which one > might follow in preparing air audit. There has 
been; however, another procedural algorithm developed by 
Halpejni (1981), and we have included it in the following pages to 
demonstrate what 4 more complex analysis of auditing might look 
like. 

Whichever/ set of procedures are chosen (of perhaps, if new 
precedures are developed), the auditing technique constitutes a 
significant addition to the methodological armory of the natural- 
istic inquirer. While audits will net deal with all trustworthiness 
questions ~ for example,, they do not^touch credibility or trans- 
ferability issues — they do provide major assurance of the de- 
pendability and confirmability of such a study. As Guba (1981) 
has suggested, while they may not* provide unassailable evidence 
with respect to these criteria, they do contribute significantly to 
the establishment of plausibility. 
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Algorithm of an audit procedure 
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Hake judjocne/ 
Draw conclusion. 



Describe state- 
ments tn report. 
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No 



Describe gaps 
in reporc. 



(from Halpern, 1981, pp.\S5-61). 
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